Systems and methods for micro-credential accreditation

ABSTRACT

Systems and methods provide micro-credential accreditation. The systems and methods analyze, using one or more prediction models, received text submissions received from applicants via interaction with an applicant device. The prediction model(s) fit one or more micro-credentials to the received text submission, which may collectively or independently qualify the applicant for one or more accreditation credits. By processing the received text submission, the systems and methods allow for consistent and standard output of micro-credentials by the prediction model(s). Furthermore, the systems and methods provide for monitoring the prediction model output(s) to ensure ethical fairness across varying demographic groups of applicants.

RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Ser.No. 63/087,072, entitled “System and Method for Micro CredentialAccreditation”, filed on Oct. 2, 2020, and which is incorporated hereinby reference in its entirety.

BACKGROUND

As the result of a long and complex history of racism, classism andoppression, minority individuals lag behind their majority peers inacademic achievement, career readiness, and career enhancement. This laghas been influenced by structural inequalities such as those framed bypoverty, migration, and/or undocumented status or homelessness. (Massey,D. S., & Denton, N. A. (1993). American apartheid: Segregation and themaking of the underclass. Harvard University Press). This lag has alsobeen called an opportunity gap because, in part, minoritized individualslack access to the majority peers' experiences and discourses upon whicheducational and professional achievement is defined. (Carter, P. L., &Welner, K. G. (Eds.). (2013). Closing the opportunity gap: What Americamust do to give every child an even chance. Oxford University Press). Assuch, the lived experiences of minoritized individuals are often definedin terms of deficits and do not carry the cultural capital found in moreprivileged students' experiences (AP courses, early college, fundedextracurricular experiences, gap year opportunities). Yet, manyminoritized individuals bring a unique set of life experiences (e.g.,immigrant families, resilience, perseverance, stress management, andconflict resolution) that have provided them with the skills andcompetencies that are key outcomes that could provide alternativeaffordances (e.g., bilingualism, problem-solving, critical thinking,collaboration). The embodiments discussed herein turn the unequalbackgrounds and social circumstances that are part of many minoritizedindividuals' lives into assets through a process of drawing upon anddocumenting their lived experiences within the language of educational,social, and cultural capital. (Darling-Hammond, L. (2015). The flatworld and education: How America's commitment to equity will determineour future. Teachers College Press).

SUMMARY

The systems and methods herein recognize that unequal backgrounds andsocial circumstances are part of many minoritized individuals' lives andthat these circumstances are underrepresented in credentials forapplications such as college. The systems and methods herein addressthis recognition by turning these backgrounds and circumstances intoassets through a process of drawing upon and documenting their livedexperiences within the language of educational, social, and culturalcapital. Moreover, the systems and methods do so in a technical mannerthat allows for differing levels of input to be normalized so that eachbackground and circumstance can appropriately, and ethically, be turnedinto a useable credential.

In this disclosure, lived experiences (LivedX) are understood from aphenomenological perspective and through a Funds of Knowledge framework.Phenomenology allows for an exploration of Lived Experiences as afunction of being in the world, encounters with others, temporality,spatiality, and a focus on personal existence. (Heidegger, M. (2008).Being and Time. New York, N.Y.: Harper Perennial). The below-discussedframework provides a way for situating personal existence within asociocultural and sociopolitical framework as “historically accumulatedand culturally developed bodies of knowledge and skills for household orindividual functioning and well-being.” (Moll, L. C., Amanti, C., Neff,D., & Gonzalez, N. (1992). Funds of knowledge for teaching: Using aqualitative approach to connect homes and classrooms. Theory intopractice, 31(2), 132-141). Peoples' lives are not blank slates and theirLived Experiences are unique and different for each individual whilebeing central to their existence. Furthermore, Lived Experiences arerich in meaning and imbued with skills and competencies and recognizing,validating, and valuing students' Lived Experiences can play animportant role in bridging the opportunity gap in educationalachievement, career pathways and enhancement.

Every day we develop a skill or up skill ourselves by doing basic tasksat home, school, work, internships, apprenticeships, and volunteeractivities. These skills do not receive their due credit in educationaland workplace settings. Something as small as planning a trip over theweekend involves many skills such as time management, financialmanagement, collaboration, communication, co-ordination and many more.Until now these life skills also known as soft skills, highly valuableskills, marketable skills, essential life skills, and/or power skills(individually and collectively referenced herein as “skills”) have notbeen recognised. The systems and methods disclosed herein allow fordocumentation of individuals' life experiences (also known as livedexperiences). However, mere documentation has limitations in that suchdocumented life experiences must be appropriately validated and assigneda credential amount. The systems and methods apply technical AI andMachine learning to validate these skills embedded in each experience,issue micro-credentials, and accredit them in a manner that istrustworthy, and useable without breaching ethical considerations.

The present disclosure describes a skills documentation system thatbridges the gap between individual users, educational organizations, andcompanies to assist with college admissions, new employee recruitment,career pathways, and career enhancement. The system can be used byindividuals to match their skills with ideal career fields, connect withlearning content, and peer and adult mentors. The system employseducation and psychology frameworks to create a skills profile toprovide additional data for college admissions, career recommendations,and job recruitment for employers. The systems and methods hereinutilize an online portal that translates peoples' everyday lifeexperiences into trusted credentials using research frameworks andproprietary Machine Learning technology. The systems and methods resultin a portfolio of highly valued skills sought and recognized and trustedby educational institutions and workplaces. The systems and methodsherein focus on amplifying skills and talents not often recognized intraditional admission or hiring processes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a system for micro-credential accreditation, inembodiments.

FIG. 2 shows the data ingestion module of FIG. 1, in further detail.

FIGS. 3-10 show example prompts presented to applicant via applicantdevice of FIG. 1 to obtain a text submission, in embodiments.

FIG. 11 shows an example method for requesting and/or receiving textsubmission data for converting to a micro-credential, in embodiments.

FIG. 12 shows a method for converting compiled data into a pre-processedsubmission ready for classification by a micro-credential predictivemodule, in an embodiment.

FIG. 13 illustrates PAM structure in contrast with LDA and multinomialDirichlet models.

FIG. 14 shows a method for training the predictive model of FIG. 1, inembodiments.

FIG. 15 shows a method for classifying a submission to output one ormore micro-credentials, in an embodiment.

FIG. 16 illustrates a functional block diagram of a model calibrationmethod to maintain a prediction model for micro-credentialaccreditation, in embodiments.

FIG. 17 shows an example micro-credential display, in embodiments.

FIGS. 18-21 show example displays for filtering previously submittedtext submissions for micro-credential accreditation, in embodiments.

DETAILED DESCRIPTION

FIG. 1 shows a system 100 for micro-credential accreditation, inembodiments. System 100 includes an accreditation portal 102 that isaccessible by one or more applicants 104 (e.g., people desiring to havemicro-credentials, or other life experiences accredited), accreditors106 (e.g., persons, or entities such as universities, workplaces, etc.that are providing credit for the applicants' lived experiences), andadministrators 108 (e.g., personnel that manage the accreditation portal102, which may be the same as or different than the accreditors 106).

Accreditation portal 102 may be a server, or other external orcloud-hosted system, that provides functionality discussed herein. Thus,accreditation portal 102 may host, or otherwise control, web-basedaccess portal that the applicant 104, accreditors 106, andadministrators 108 access via computing devices 110. Computing devices110 may be any one or more of a laptop computer, desktop computer, smartphones, tablets, and other devices capable of accessing the web browser,or other portal, hosted by accreditation portal 102. Computing device110(1) may be referred to herein as applicant device 110(1). Computingdevice 110(2) may be referred to herein as accreditor device 110(2).Computing device 110(3) may be referred to herein as administratordevice 110(3).

Accreditation portal 102 includes one or more processors 112 and a datastore 114 that stores necessary data to implement the functionality ofone or more of a data ingestion module 116, a micro-credentialpredictive module 118, and generated micro-credentials 122 andassociated accreditation credits 123 (which may be one or more combinedmicro-credentials 122 that collectively qualify the applicant 104 for agiven accreditation credit 123, as configured by the administrator 108or accreditor 106 interacting with their given computing device 110 toset user-specific configuration settings of the accreditation portal102). The processor 112 may be any type of circuit or integrated circuitcapable of performing logic, control, and input/output operations. Forexample, the processor 112 may include one or more of a microprocessorwith one or more central processing unit (CPU) cores, a graphicsprocessing unit (GPU), a digital signal processor (DSP), afield-programmable gate array (FPGA), a system-on-chip (SoC), amicrocontroller unit (MCU), and an application-specific integratedcircuit (ASIC). The processor 112 may also include a memory controller,bus controller, and other components that manage data flow between theprocessor 112, data store 114, and other components connected to theprocessor 112.

The data store 114 (also referred to as memory) includes one or morememory allocations that store machine-readable instructions that, whenexecuted by the processor 112, control the portal 102 to implement thefunctionality and methods described herein. Said machine-readableinstructions, when executed by processor 112, implement the various“modules” discussed herein. Data store 114 additionally stores necessarydata to implement the functionality and methods described herein.

Accreditation portal 102 may interface with devices 110 to ingestnecessary data (e.g., text describing lived experiences submitted to theportal by the applicant 104), and display the associatedmicro-credentials 122 and associated accreditation credits 123. Thelived experience texts are collected via the devices 110(1) detectinginteraction by the applicant 104 with the hosted platform byaccreditation portal 102 via applicant device 110(1). Various API callsand responses are utilized between the accreditation portal 102 and thedevice 110(1) to receive the necessary information.

FIG. 2 shows the data ingestion module 116 of FIG. 1, in further detail.Applicant 104 interacts with device 110(1) to provide a lived experiencesubmission, which is received by accreditation portal 102 through one ormore API calls/responses. Raw submission 202 is stored in a data store114 of the accreditation portal 102.

FIGS. 3-11 show example prompt/response flow that is displayed on device110(1) for the applicant 104 to input a submission that is stored as rawsubmission 202, in an embodiment.

Screenshot 300, of FIG. 3, shows a screenshot where the applicant 104 isprovided with a prompt on device 110(1) to select an experience in thepre-existing categories 302 (such as, but not limited to: action,process, quality), sub-categories 304 (e.g., type of action), andlocation 306 (e.g., home, work, school, shopping, other). This allowsthe applicant 104 to identify and focus on a recent experience thatmight be of importance/value to them and they are interested indocumenting.

Screenshot 400, of FIG. 4, shows a screenshot where the applicant 104 isprovided with a prompt on device 110(1) to select additional peopleinvolved in the experience represented in the current submission. Sincethe applicant 104 is submitting their lived experience, the prompt inscreenshot 400 prompts applicants to indicate where the experience tookplace and who participating with them in the experience. This helps theapplicant 104 contextualize the experience they are planning to submit.This also allows applicant 104 to submit a diversity of experiences interms of location and various aspects of their life.

Screenshot 500, of FIG. 5, shows a screenshot where the applicant 104 isprovided with a prompt on device 110(1) to select provide a textdescription of the lived experience. This prompt (describe what happenedin this experience) sets the stage for the user for submitting theexperience. By asking them to describe the experience (thoughts andfeelings), the prompt in screenshot 500 narrows down or make theexperience focused on specific events (rather than a sequence of events)and allows the accreditation portal 102 to be able to interpret theapplicant's 104 understanding of their experience.

Screenshot 600, of FIG. 6, shows a screenshot where the applicant 104 isprovided with a prompt on device 110(1) to select provide additionaltext description of the lived experience. This prompt (what did youlearn from this experience) is used to help the applicant 104concretized their actions (e.g., in difficult situations) and use theexperience as a learning tool from the past experience as well movingforward.

Screenshot 700, of FIG. 7, shows a screenshot where the applicant 104 isprovided with a prompt on device 110(1) to select provide additionaltext description of the lived experience. This prompt (what would you dodifferently) is used to help the applicant 104 describe what and howmuch they learned from the experience, be it whether they did the rightthing in their view (and wouldn't do anything differently), or didsomething wrong and would change how their actions in similarsituations.

Screenshot 800, of FIG. 8, shows a selection prompt requesting theapplicant 104 to select a content area (e.g., category) to which thesubmitted text applies. Screenshot 800 is in an example of AP coursesfor college credit, however other categories may be prompted dependingon the applicant 104 and target micro-credentials. Screenshot 900, ofFIG. 9, shows alternate categories provided in the content-area prompt.Additional categories include engineering, business, environmentalliteracy, arts/culture, computer science, science, mathematics,literature, management, architecture, global issues, languages,healthcare, agriculture/farming, technical education, education, sports,media, peer mentoring, other.

Screenshot 1000, of FIG. 10, shows a selection prompt requesting theapplicant 104 to provide evidence (or indicate they could provideevidence) of the submitted experience. This allows the portal 102 tohave an understanding of the veracity of the submitted experience.

FIG. 11 shows an example method for requesting and/or receiving, fromapplicant 104, data for converting to a micro-credential, inembodiments. Method 1100 is performed using system 100, such asexecution of the data ingestion module 116, for example.

In block 1102, the method 1100 requests/receives an indication of aselected category associated with a submitted data. In one example ofblock 1102, the data ingestion module 116 implements one or more APIcalls/responses to/from the user device 110(1) to receive a selectedcategory (e.g., drop-down menu 302 of FIG. 3).

In block 1104, the method 1100 requests/receives an indication of anassociated event that occurred that is associated with a submitted data.In one example of block 1104, the data ingestion module 116 implementsone or more API calls/responses to/from the user device 110(1) toreceive a selected action (e.g., drop-down menu 304 of FIG. 3).Selectable choices presented to the Applicant 104 (e.g., via device110(1)) may be pre-determined based on the selection in step 1102.

In block 1106, the method 1100 requests/receives an indication of alocation where the associated event occurred and is associated with asubmitted data. In one example of block 1106, the data ingestion module116 implements one or more API calls/responses to/from the user device110(1) to receive a selected location (e.g., selection list 306 of FIG.3). Selectable choices presented to the Applicant 104 (e.g., via device110(1)) may be pre-determined based on the selection in one or both ofstep 1102 and 1104.

In block 1108, the method 1100 requests/receives an indication of otherpersons that were involved in the associated event and is associatedwith a submitted data. In one example of block 1108, the data ingestionmodule 116 implements one or more API calls/responses to/from the userdevice 110(1) to receive a selected action (e.g., selection list 402 ofFIG. 4). Selectable choices presented to the Applicant 104 (e.g., viadevice 110(1)) may be pre-determined based on the selection in one ormore of steps 1102-1106.

In block 1110, the method 1100 requests/receives a text description ofthe associated event and is associated with a submitted data. In oneexample of block 1110, the data ingestion module 116 implements one ormore API calls/responses to/from the user device 110(1) to receive atext description including one or more of the scenario/event thatoccurred, personal experience in the scenario/event, actions theapplicant 104 engaged in, and lessons learnt during the scenario/event(e.g., input to the prompts shown in FIGS. 5-7). Selectable choicespresented to the Applicant 104 (e.g., via device 110(1)) may bepre-determined based on the selection in one or more of steps 1102-1106.

In block 1112, the method 1100 requests/receives an indication ofavailable evidence, or the evidence itself, of the associated event andis associated with a submitted data. In one example of block 1112, thedata ingestion module 116 implements one or more API calls/responsesto/from the user device 110(1) to receive an indication of availableevidence, or the evidence itself (e.g., response to prompt shown in FIG.10).

The responses received in blocks 1102-1112 may be saved collectively orindividually as raw submission 202 of FIG. 2. In block 1114, the method1100 may compile the responses into a complied submission 204. Referringback to FIG. 2, the raw submission 202 may be a series of data responsesreceived in response to any one or more of the above-discussed prompts.These responses are then, by the data ingestion module 116, formattedinto a compiled submission 204. Compiled submission 204 takes thedisparate responses to the prompts and puts them into a text stringdescribing the input experience. In the text string of the compiledresponse 204, prompts that have a selectable answer (e.g., 302, 304, 306of FIG. 3) may be compiled into a first portion of the text string, andprompts that have a text-input response (e.g., input 502, 504 of FIG. 5)may be included as-is as a second portion of the text string. Below isan example of submitted narrative:

-   -   This experience took place at my [Other]. I was with some        Strangers. I was at the DMV and saw someone trying to ask for        the bathroom in ASL. I knew what they were saying so I stood up        and helped them find the bathroom. I learned that it does not        matter where you are and what you are doing, helping someone who        needs it is important. I would not do anything different next        time.        “This experience too place at my [other].” is generated in        response to a selection in input drop-down 302. “I was with some        Strangers.” is generated in response to selection of box 402 of        FIG. 4.

Auto-generating the narrative, as opposed to allowing the user to simplyinput a narrative and output what the user inputs, allows for aconsistently formatted narrative for review. Thus, each user's livedexperience can be accurately analyzed by the back-end system, or areviewer at a work, college, school, etc., to provide an appropriate oneor more micro credentials therefore.

Data ingestion module 112 may further process the compiled data 204(e.g., after implementation of method 1100), to generate a processedsubmission 206. The data ingestion module 112 may generate processedsubmission 206 by processing the compiled text data 204 to prepare forclassification by the predictive model 120 of micro-credentialpredictive module 118 (FIG. 1) to generate one or more micro-credentials122 which may, independently or in combination, qualify the applicant104 for one or more accreditation credits 123. To generate the processedsubmission 206, the data ingestion module 112 may remove stop wordswithin the compiled data 204, implement stemming on the compiled data204, implement lemmatization on the compiled data 204, convert thecompiled data 204 to a plurality of N-Grams, covert the compiled data204 (after any one or more of the above pre-processing techniques) intothe term-frequency inverse document frequency (tf-idf) matrix.

FIG. 12 shows a method 1200 for converting the compiled data 204 into apre-processed submission 206 ready for classification by themicro-credential predictive module 118, in an embodiment. Method 1200 isimplemented by the data ingestion module 112, for example.

In block 1202, gibberish text within complied data 204 is removed. Inone example of block 1202, the data ingestion module 112 analyzes thecomplied data 204 and removes text that is incoherent, and unable to becorrected via spell-check, or other correction methods.

In block 1204, the remaining complied data 204 is standardized. In oneexample of block 1204, the data ingestion module 112 analyzes theremaining complied data 204 after block 1202 and standardizes the text.The text may be standardized by performing spell check, removingredundant and/or duplicate words, etc.

In block 1206, whitespace within complied data 204 is removed. In oneexample of block 1206, the data ingestion module 112 analyzes thecomplied data 204 and removes whitespace that is within, before, orafter the text portion of the submission.

In block 1208, stop-words within the remaining complied data 204 areremoved. In one example of block 1208, the data ingestion module 112analyzes the remaining complied data 204 and removes stop-words therein.Extremely common words in text which would appear to be of little valuein helping select micro-credentials matching the submitted experience bythe applicant 104 are excluded from the vocabulary entirely. These wordsare referred to herein as “stop words.” Stop words include, but are notlimited to, words such as “the”, “is”, “are”, and so on. The generalstrategy for determining a stop word list is to sort the terms byoccurrence frequency in a particular experience text document, and thento take the most frequent terms, often hand-filtered for their semanticcontent relative to the domain of the documents being indexed, as a stoplist, the members of which are then discarded during indexing. In somenatural language processing applications stop word removal will havevery little impact on its predictive performance, rather it reducescomputational complexity of training the model.

In block 1210, stemming on the remaining complied data 204 isimplemented. In one example of block 1210, the data ingestion module 112analyzes the remaining complied data 204 after block 1208 and implementsstemming. Stemming may be performed after other blocks within method1200 without departing from the scope hereof. Stemming is used ininformation retrieval systems to make sure variants of words are notleft out when text is retrieved (Julie Beth Lovins. 1968. Development ofa stemming algorithm. Mech. Transl. Comput. Linguistics,11(1-2):22-31.). The process is used in removing derivational suffixesas well as inflections (i.e. suffixes that change the form of words andtheir grammatical functions) so that word variants can be conflated intothe same roots or stems. For example, words like “playing”, “plays”,“played” have the same common root “play”. Therefore, by stemming wetransform all the word derivations into their corresponding root. Italso reduces computational complexity of the process, and increasepredictive performance of the model. In one embodiment, blcok 1210implmeents Porter's stemmer. (Porter, M. F., 1980. An algorithm forsuffix stripping. Program; which is incorporated by reference herein).

In block 1212, lemmatization on the remaining complied data 204 isimplemented. In one example of block 1212, the data ingestion module 112analyzes the remaining complied data 204 after block 1210 and implementslemmatization. Lemmatization may be performed after other blocks withinmethod 1200 without departing from the scope hereof. Lemmatization putsemphasis on vocabulary and morphological analysis of word and tries toremove inflectional endings, thereby returning words to their dictionaryform. Lemmatization checks to make sure that words are properly used intext. For example, it analyzes if query words are used as verbs ornouns.

In block 1214, a plurality of N-grams is generated from the remainingcomplied data 204 is implemented. In one example of block 1214, the dataingestion module 112 analyzes the remaining complied data 204 afterblock 1212 and generates a plurality of N-grams. N-gram generation maybe performed after other blocks within method 1200 without departingfrom the scope hereof. In embodiments that optionally include block1214, instead of working with single words to construct the vocabularyin a Bag-of-word model, N-grams are phrases constructed with consecutiveN words in the source text. Each of the N-gram can then be considered aterm in the text retrieval and analysis.

In block 1216, a Term Frequency Inverse Document Frequency (TF-IDF)Matrix is generated from the remaining complied data 204. In one exampleof block 1216, the data ingestion module 112 analyzes the remainingcomplied data 204 after block 1214 and generates a TF-IDF Matrix. TF-IDFMatrix generation may be performed after other blocks within method 1200without departing from the scope hereof. The TF-IDF scoring algorithmconsiders frequency of the terms appearing in a document, and at thesame time put more weight on those terms that occur less frequentlyacross documents in the text corpus. The term frequency, tf_(t,d)describes how frequent the term t is in a given document d, and isexpressed as log normalized as Log(1+tf_(t,d)). On the other hand, dfrefers to document frequency and relates to the number of document thatcontains the search keyword. The inverse document frequency (idf)describes the relevance of the search term in relation to all thedocuments in the collection, as depicted in the following equation:

$\begin{matrix}{{idf}_{t} = {\log\frac{N}{{df}_{t}}}} & {{Equation}\mspace{14mu}(1)}\end{matrix}$

where N is the total number of documents in the text corpus. The(tf−idf) is therefore defined as:

$\begin{matrix}{{tfidf} = {{\log\left( {1 + {tf}_{t,d}} \right)}\log\frac{N}{{df}_{t}}}} & {{Equation}\mspace{14mu}(2)}\end{matrix}$

The processed submission 206 may then be analyzed by micro-credentialpredictive module 118 via classification of the processed submission 206by the predictive model 120. In one example, the predictive model 120 isbased on one or more of a Pachinko Allocation Model (PAM), a LatentDirichlet Allocation (LDA) Model, and a MLkNN model.

Topic modeling discovers the thematic structure of a given text corpus.It models the relationships between the vocabulary of the corpus and thedocuments through the thematic structure. Topic models gained popularityin recent years as the learned structure and relationships can be usedfor the analysis of large-scale natural language text data, includingquery, discover trends, language translation, document classification,information retrieval, text summarization, sentiment analysis andvarious data mining problems. Given a corpus of text documents,parameter estimation in these topic models extracts a lower dimensionalset of multinomial, i.e., multi-label word distributions called topics.Mixtures of the topics provide high probability to the training data,and the highest probability words in each topic provide the keywordsthat briefly summarize the themes in the text corpus (e.g., theprocessed submission 206).

The topic modeling problem can be attempted as a multi-labelclassification problem, where each document may belong to severalpre-defined topics simultaneously. The problem can be formally definedhere. Let X denote the domain of documents and let Y={1, 2, . . . , Q},be the finite set of topics, i.e., the labels. Given a training set,T={(x_(i), Y_(i))}_(i=1) ^(m), where x_(i)∈X, Y_(i)∈ drawn from anunknown distribution D, the objective is to generate a multi-labelclassifier, h: X→2^(Y), which optimizes some specific evaluation metric.Instead of providing only the input to output label mapping, thelearning algorithm will produce a real-valued function of the form,ƒ:X×Y→

. It is assumed that, given a document, x_(i), and its associated topiclabel set, Y_(i), a successful learning system is going to return largervalues for labels in Y_(i) than those not in Y_(i), i.e.,ƒ(x_(i),y₁)>ƒ(x_(i), y₂), for any y₁∈Y_(i) and y₁ ∉Y_(i).

FIG. 13 illustrates the PAM structure in contrast with LDA andmultinomial Dirichlet models. In FIG. 13, the four topic modelstructures include: (a) Dirichlet multinomial: for each document, amultinomial distribution over words is sampled from a single Dirichletdistribution. (b) LDA: it samples a multinomial over topics for eachdocument, and then generates words from the topics. (c) a 4-level PAM:it contains a root, a set of super-topics, a set of sub-topics and aword vocabulary. Both the root and the super-topics are associated withDirichlet distributions, from which multinomials over their children foreach document are sampled. (d) PAM: an arbitrary directed acyclic graph(DAG) structure encoding the topic correlations. Each interior node isconsidered a topic and associated with a Dirichlet distribution

The LDA2vec model learns dense word vectors jointly with Dirichletdistributed latent document-level mixtures of topic vectors (ChristopherE Moody. 2016. Mixing dirichlet topic models and word embeddings to makelda2vec.arXiv preprint arXiv:1605.02019.). In one embodiment, thepredictive model 120 used by micro-credential predictive module 118 isbased on Pachinko Allocation Model instead of the LDA as the algorithmimproves upon LDA based approaches by modeling correlations betweentopics in addition to the word correlations which constitute topics. PAMmay be advantageous, in certain embodiments, as it provides moreflexibility and greater expressive power than Latent Dirichletallocation.

In one embodiment, the predictive model 120 used by the micro-credentialpredictive module 118 is trained based on a Pam2vec algorithm. In thismodel, the total loss term, L is the sum of the Skipgram NegativeSampling Loss (SGNS), L_(ij) ^(neg) with the addition of aPachinko-likelihood term over document weights, L^(d). The loss isconducted using a context vector, {right arrow over (c_(j))}, pivot wordvector, {right arrow over (w_(j))}, target word vector w_(i), andnegatively-sampled word vector w_(l). Pairs of pivot and target words(j, i) are extracted when they co-occur in a moving window scanningacross the corpus (e.g., processed submission 206). For everypivot-target pair of words the pivot word is used to predict the nearbytarget word. Each word is represented with a fixed-length densedistributed-representation vector. The same word vectors are used inboth the pivot and target representations. Both words and documentvectors are then embedded into the same space and the predictive model120 is trained based on both representations simultaneously. By addingthe pivot and document vectors together, both spaces are effectivelyjoined.

FIG. 14 shows a method 1400 for training the predictive model 120 ofFIG. 1, in embodiments. Method 1400 is implemented by accreditationportal 102, such as via a micro-credential predictive module 118, in anembodiment.

In block 1402, method 1400 receives training data. Training data may bea plurality of the received processed submissions 206, or may be createdones thereof that are associated with appropriate micro-credentials.

In block 1404, method 1400 calculates a total loss term of the trainingsample received in block 1402. In one example of block 1404,micro-credential predictive module 118 calculates the total loss term,L, as the sum of the Skipgram Negative Sampling Loss (SGNS), L_(ij)^(neg) with the addition of a Pachinko-likelihood term over documentweights, L^(d). The loss is conducted using a context vector, {rightarrow over (c_(j))}, pivot word vector, {right arrow over (w_(j))},target word vector w_(i), and negatively-sampled word vector w_(l).

In block 1406, the method 1400 extracts pivot and target words of saidtraining sample, and represents them with a fixed-length dense vectors.In one example of block 1404, micro-credential predictive module 118extracts pairs of pivot and target words (j, i) when they co-occur in amoving window scanning across the corpus (e.g., said given sampleprocessed submission 206). For every pivot-target pair of words thepivot word is used to predict the nearby target word. Each word isrepresented with a fixed-length dense distributed-representation vector.

In block 1408, the method 1400 embeds the extracted words and vectorstogether in a single embedded space, and trains a predictive model usingthe embedded space.

In block 1410, the method 1400 outputs the trained predictive model. Inone example of block 1410, micro-credential predictive module 118outputs predictive model 120 of FIG. 1.

As discussed above, the trained predictive model 120 is used tocharacterize one or more micro-credentials 122 to a submission receivedfrom applicant 104. The predictive model 120 may be based on a singlepredictive algorithm (e.g., the Pam2vec algorithm discussed above withrespect to FIG. 14), or a plurality of predictive algorithms.

FIG. 15 shows a method 1500 for classifying a submission (e.g.,processed submission 206) to output one or more micro-credentials (e.g.,micro-credentials 122 of FIG. 1), in an embodiment.

In block 1502, the method 1500 receives a submission for analysis. Inone example of block 1502, the micro-credential predictive module 118receives the processed submission 206 from the data ingestion module116.

Method 1500 then implements one or more of branches 1504, 1506, and1508.

Branch 1504 implements a ML-kNN predictive model (which is an example ofpredictive model 120). ML-kNN is an effective realization of themultilabel classification algorithm (Min-Ling Zhang and Zhi-Hua Zhou.2007. Ml-knn: A lazy learning approach to multi-label learning. Patternrecognition, 40(7):2038-2048, which is incorporated herein byreference.). For each unseen document instance, its k nearest neighborsin the training set are first identified (blcok 1510). Subsequently,based on assessment of information gained from the predicted label setsof these neighboring instances, i.e., the number of neighboringinstances (block 1512) belonging to each possible class, maximum aposteriori (MAP) principle is employed (block 1514) to determine thelabel set for the unseen instance (block 1516).

Branch 1506 implements a latent dirichlet allocation predictive model(which is an example of predictive model 120). Latent DirichletAllocation (LDA) is a topic model applied mostly to generate texts basedon topic of choice (David M Blei, Andrew Y Ng, and Michael I Jordan.2003. Latent dirichlet allocation. Journal of machine Learning research,3(January):993-1022; which is incorporated by reference herein.). Italso can be employed in the text categorization problems. The receivedsubmission is split into a plurality of sequential features (block1518). LDA is then implemented (block 1520) which represents eachdocument as a mixture of topics, where each topic is a multinomialdistribution over words with respect to a vocabulary. To generate adocument, the method 1500 LDA first samples a per-document multinomialdistribution over topics from a Dirichlet distribution. Then itrepeatedly samples a topic from this multinomial and samples a word fromthe topic. Topics extracted (block 1522) by LDA capture correlationsamong words. An output prediction vector of the given topic to a set ofpotential micro-credentials is calculated (block 1524) for eachgenerated topic, and those satisfying a threshold value are indicated asappropriate micro-credentials.

Branch 1508 implements a pachinko dirichlet allocation predictive model(which is an example of predictive model 120). The Pachinko AllocationModel (PAM) utilizes a multi-level directed acyclic graph (DAG)structure to model the topic correlations and hierarchies, where theleaf nodes represent the words, and the nodes in the inner levelsrepresent topics (Wei Li and Andrew McCallum. 2006. Pachinko allocation:Dag-structured mixture models of topic correlations. In Proceedings ofthe 23rd international conference on Machine learning, pages 577-584;which is incorporated by reference herein). The received submission issplit into a plurality of sequential features (block 1526). Then, toimplement PDA model (block 1528), topics are considered distributionover words in the vocabulary. It samples a topic path for each word toidentify the topic associations. The model is trained with Gibbssampling method. Topics extracted (block 1530) by PAM capturecorrelations among words. An output prediction vector of the given topicto a set of potential micro-credentials is calculated (block 1532) foreach generated topic, and those satisfying a threshold value areindicated as appropriate micro-credentials.

Although only three branches (branches 1504, 1506, and 1508) are shown,there may be more or fewer branches without departing from the scopehereof. Moreover, different predictive models may be used other thanthose shown. For example, other types of applicable predictive modelsinclude, but are not limited to, logistic regression, support vectormachines, and Nai{umlaut over (v)}e Bayes. As an example, there may be aset number of target (potential) micro-credentials 124 (e.g., 152potential assignable micro-credentials). Further, the potentialassignable micro-credentials may be separated into a hierarchy (e.g.,three levels: level 1, level 2, and level 3). Each targetmicro-credential 124 may be associated with a target classifier 126(which may be any one or more of the branches discussed in method 1500).Certain target micro-credentials 124 may be available only depending onspecific responses to the prompts discussed in FIGS. 3-11. If applicantsays they would not do anything differently in response to the “movingforward” prompt shown in FIG. 7, a “reflection” micro-credential may notbe available. In one embodiment, an individual binary classifier may befit to each micro-credential target (e.g., using logistic regression,per target). These binary classifiers may then be used (in addition toor in alternative to branches 1504, 1506, and 1508) to classifymicro-credentials to a received submission.

Referring back to FIG. 15, in block 1534, method 1500 applies ensemblemulti-label learning. In one example, multiple different learningalgorithms must come to a consensus on target potentialmicro-credentials associated with the received submission by eachbranch. In another example, the associated prediction vectors must reacha desired threshold. Each of the three branches (1504, 1506, 1508) maybe separate multi-label (i.e., multi-output) classifiers thatindependently predicts the micro-credentials for a given submission(1516, 1524, 1532). The ensemble multi-label learner (1534) receives thepredictions from the three branches and combines the predictions,through one or both of Boosting and stacked generalization and generatesthe final consensus prediction of micro-credentials.

In block 1536, the target potential micro-credentials that arecollectively agreed upon in block 1534 (or those output by one or morebranches if block 1534 is not included) are output as assignedmicro-credentials (e.g., micro-credentials 122) and associated with thereceived submission. Block 1536 may include identifying a plurality ofthe assigned micro-credentials 122 that collectively qualify theapplicant 104 for one or more accreditation credits 123 based onconfiguration settings defined by accreditor 106 interaction withaccreditor device 110(2). Block 1536 may include transmitting an APIcall to display one or more of the assigned micro-credentials 122 andassociated accreditation credits 123 on the applicant device 110(1)and/or accreditor device 110(2).

Model Maintenance:

Machine learning models are inherently sensitive to the distribution ofthe training dataset. The population data distribution may change afterthe model is built on a subset obtained at a specific timestamp in thepast. If it happens, the model will suffer consequences like increasingin the misclassification rate, losing reliability. Therefore,post-training model maintenance and calibration is a necessary stepwhich needs to be scheduled on a regular basis.

The following illustrates how annotated data (e.g., training data usedto create and/or update the prediction model 120) impacts correctness,and possibly inherent biases implemented by the prediction model 120. Tocreate initial training data to train the prediction model 120 for use,a training phase was initiated that focused on creating a process forestablishing interrater reliability (IRR) using the educational researchliterature. The IRR process included the following steps: 1) training adiverse group of data taggers as qualitative data labelers/coders usingthe accreditation platform 102; and 2) using accreditation platform 102driven a priori labels/codes to analyze submitted narratives (e.g.,submissions 202). This training phase methodology led to outlining theprocedure to establish IRR using a priori codes. The coded data willthen be used to develop a second phase methodology in which a machinelearning algorithm will be created.

According to Walter et al.'s 1998 power calculations for interraterreliability, a minimum of 2 raters are needed for 43 pieces ofinformation to have enough power to find Interclass CorrelationCoefficient (ICC) agreement above 0.70. Since we wanted to compare preand post, 4 raters instead of only 2 ensures more power for comparisons(i.e. 2 raters×2 attempts) and the same rationale was used as we wantedto compare 3 frameworks (2 raters×3 frameworks). Therefore 6 raters arerecommended for coding. Additionally, if we wanted to conduct t tests bypre-post or by “correct” answer for each code, a power analysis usingG*Power software shows that 42-45 items are needed. As we expect a fewproblematic observations, 45 items/observations are recommended.

TABLE 1 Pre and post coding results Pre-Training Post-TrainingConsidering the Items Average Correct Answers 59% 83% Average Agreement(not necessarily 59% 70% correct) Number of Items Matched Correct & 21(47%) 22 (50%) Agreement percentages ICC for the Measure 0.46 0.83Considering the Coders Coder 1 Correctness 53% 80% Coder 2 Correctness56% 81% Coder 3 Correctness 58% 77% Coder 4 Correctness 64% 89% Coder 5Correctness 67% 84% Coder 6 Correctness 53% 82%

We then ran statistical analysis to measure Intraclass Correlation at95% confidence interval and share the results in Table 2:

TABLE 2 Average total agreement = 80% Intraclass Correlation Coefficient95% Confidence Interval F Test with True Value Intraclass Lower Upper 0Correlation^(b) Bound Bound Value df1 df2 Sig Single Measures .477^(a).397 .561 6.462 119 595 .000 Average Measures .845^(c) .798 .884 6.462119 595 .000

Table 2 establishes that two-way mixed effects model where peopleeffects are random and measures effects are fixed. It is assumed thatthe estimator is the same, whether the integration effect is present ornot. Type C intraclass correlation coefficients using a consistencydefinition. The between-measure variance is excluded from thedenominator variance. The estimate is computed assuming the interactioneffect is absent, because it is not capable of estimation otherwise. Thecalculation shown in table 2 provides for a high confidence in datacoding, which is then used to code a larger set of data for preparingthe prediction model 120.

One simple solution, where the prediction model is biased due to poordata labeling during training would be to retrain the model with the newtraining dataset when available. But, the approach may become timeconsuming and model still may suffer the aforementioned issues duringthe retraining interval. The second approach, is to apply incrementalupdate on the model parameters after each new training sample wheneverpushed from the platform.

FIG. 16 illustrates a functional block diagram of a model calibrationmethod 1600 with a flow diagram which depicts all the modules necessaryto maintain the model (e.g., predictive model 120) deployed inproduction. Method 1600 is implemented using the model maintenancemodule 128, for example. The method 1600 implements of three components:i) Aggregator 1602, ii) Validator 1604, and iii) Model comparator 1606.

A new sample 1608 is received and stored within a buffer 1610 of size msamples. Method 1600 then re-computes the statistics of the m samplesincluding the new sample 1608. It reports to the validator 1604 when thebuffer 1610 is full. The pipeline utilizes an adaptive algorithm tochange the size of buffer 1610 in each run based on the varianceobserved in the statistics.

The validator 1604 employs an ensemble machine learning approach todetect if any of the new samples 1608 present in the buffer 1610 is anoutlier. If an outlier is present, it re-assesses the statisticscalculated by the aggregator module 1602 and then triggers theincremental model parameter update (block 1612) on a model clone 130.The updates do not affect the model being used in the productionenvironment (e.g., predictive model 120). Incremental update steps varysignificantly by model types, model complexity and difference betweenthe previous and new statistics. The validator module 1604, therefore,needs to account for successfully deciphering and understanding thegeneral trends of the data and semantics of the population statistics.Method 1600 utilizes multifarious techniques involving machine commonsense reasoning, rare event modeling and model selection from anensemble to address the changes.

The model comparator 1606 then performs an evaluation (block 1614) usinga pre-determined validation data to compare model performances of theproduction environment (e.g., predictive model 120) and the newlyupdated model clone 134. If the predictive performance of the updatedmodel clone 134 is satisfactory, it triggers a backup and deploy eventto replace the production environment model (e.g., predictive model 120)in production with a model update 132.

Ethical Awareness Calibration

Model maintenance module 128 may further analyze the outputs of themodels and verify that the models are performing in an ethical manner.The purpose of the embodiments of the systems and methods describedherein is to improve outcomes for minoritized students, butunintentional and unwanted bias may occur in machine learning framework.Model maintenance module 128 may compensate for this unintentional andunwanted bias by monitoring the output and verifying it is performingappropriately.

Machine learning models (e.g., predictive models 120) may be inherentlybiased if the data to which they are trained on has said inherent bias(see my comment above). The training data is annotated using humanannotators, and thus the annotators shape the data used by the model.Model maintenance module 128 may analyze the output of the models 120and provide valuable feedback to our human annotators. This feedbackprovides annotators the opportunity to reconsider their own inherentbiases and to account for the way the models leverage their annotations.The only way to achieve ethical results in use cases such as these is toview the entire process iteratively—from end to end. The annotators donot simply decide on annotations once and for all. Rather, they areprivy to the sorts of mistakes the machine learning model makes fromtheir annotations in order to consider possible biases in their ownannotation work. They consider how to address the root (rather thansuperficial) of the unwanted bias (Corbett-Davies, S. and Goel, S.(2018). The measure and mis-measure of fairness: A critical review offair machine learning. CoRR, abs/1808.00023.; Fazelpour, S. and Lipton,Z. C. (2020). Algorithmic fairness from a non-ideal perspective. AIES'20, 219 page 57-63, New York, N.Y., USA. Association for ComputingMachinery.; Estrada, D. (2020). Ideal theory in AI ethics. CoRR,abs/2011.02279.).

The alternative is to ensure that the unfairness generated by analgorithm on particular annotations is distributed equally among allgroups. This does not usually improve the situation much because itfails to account for and address the characteristics of the data thatare leading to the unwanted bias in the first place. Further, suchsuperficial alternative approaches fail to provide guidance on how themachine learning fueled platform like the systems and methods describedherein should address the root cause of the unwanted biases. To addressthese inherent bias problems, the model maintenance module 128 mayimplement a machine-in-the-loop strategy to understand the intersectionof machine and human bias for an effective and fair micro-credentialingprocess.

Referring back to FIG. 1, the model maintenance module 128 may analyzethe output micro-credentials 122 to calculate a fairness metric 134. Inone embodiment, the fairness metric 134 is based on conditionalstatistical parity (CSP). CSP measures whether particular groups ofapplicants 104 have equal probability of receiving a favorable outcome(in this case a credential 122) while permitting a legitimate factor toaffect the prediction fairly. The applicant pool considered for thefairness metric may have required thresholds, such as number ofsubmitted experiences. In one embodiment, submissions from applicants104 who have not submitted at least 5 submissions are not considered forthe fairness metric 134 because applicants 104 who have submitted fewerthan 5 experiences to be too inexperienced and exclude them fromconsideration.

Thus, to calculate the fairness metric 134, the model maintenance module128 may identify a CSP value 136 of each target micro-credential 124.The CSP value 136 is calculated by determining the probability of saidtarget micro-credential 124 being issued to a certain demographic groupof applicants 104.

As discussed above, the micro-credentials 122 may be divided into ahierarchy of levels. In testing of the present embodiments herein, forlevel-1 and level-2 credentials, CSP was found for many—but notall—credentials. We found that CSP exists for most credentials betweenmost groups, but occasionally a group achieves substantially fewercredentials than other groups. For example, the applicants 104 whoidentified as white achieved a CSP of 0.1049 for the “working withothers” credential, whereas students who identify as black or AfricanAmerican achieved a CSP of 0.0368 (see 2). If the data annotations aretaken to be authoritative (and not unintentionally inherently biased),students who identify as black or African American would necessarily besomehow deficient (perhaps due to systemic discouragement of theseexperiences). A second possibility also exists, however. Students whoidentify as black or African American may express how they work withothers in ways that are different from white students and in ways thatthe annotators might have missed. These questions cannot be answered bythe outputs of the model alone. They need to be answered by the subjectexperts (in this case, the annotators).

Accordingly, when the CSP value 136 for a first demographic ofapplicants 104 for a given target micro-credential 124 differs from asecond demographic of applicants 104 for said given targetmicro-credential 124 greater than or equal to a CSP threshold value 138,the model maintenance module 128 may trigger a model update 132. In oneembodiment, the CSP threshold value 138 is equal to or greater than 0.05as credentials of concern.

Example 1: Unassigned Micro-Credentials

The following example shows a processed submission (e.g., processedsubmission 206) that was not assigned at least 5 micro-credentials byhuman annotators (e.g., 5 of the target micro-credential 124 were notincluded as a micro-credential 122 when human annotators assignedmicro-credentials, and thus would not be used to train the predictivemodel 120). The submission text was as follows: “So, I have been workingon this project for some time and the challenge I am facing is tobalance various responsibilities—I have to manage multiple roles andvarious tasks. I have been feeling very stressed out and feeling theneed to take a break. I reached out to some family and friends and havea conversation and found a way to destress. I learnt that there are upsand down in this process and one should keep the bigger picture in mindand not let bad news get me down. I would not allow myself to go downthis path for too long and seek some interventions earlier.”

In example 1, the machine learning model (e.g., predictive model 120)predicted the following level-3 codes that the human coders missedduring annotations. The predictions were later marked approved afterreview of by annotators (e.g., administrator 108) and being included asnew training data to generate model update 132.

-   -   Recognize one's emotions, thoughts, behaviors, and/or body        sensations.    -   Uses strategy to regulate cognitive, behavioral, and/or        emotional states in the moment (e.g., reaching out to someone,        breathing, etc.)    -   1.2.3 Enacts Self and/or others advocacy    -   1.4.2 Observes situations carefully    -   6.5.2 Reflects on next steps, or what they could have done        differently based on the submitted experience.

Example 2: Unassigned Micro-Credentials

The following example shows a processed submission (e.g., processedsubmission 206) that was not assigned at least 6 micro-credentials byhuman annotators (e.g., 6 of the target micro-credential 124 were notincluded as a micro-credential 122 when human annotators assignedmicro-credentials, and thus would not be used to train the predictivemodel 120). The submission text was as follows: “I worked with my sisterto resolve a family conflict related to my mothers health. She did notdo her part and that led to my mother's deteriorating health. I learntthat I can not trust and rely on my sister. I would have found otherpeople to take care of my mom's health and manage the situation. I willidentify 2-3 resource people and keep in touch with them to make surethat this doesn't happen again.”

In example 2, the machine learning model (e.g., predictive model 120)predicted the following level-3 codes that the human coders missedduring annotations. The predictions were later marked approved afterreview by annotators (e.g., administrator 108) and are being included asnew training data to generate model update 132.

-   -   1.1.1 Recognize one's emotions, thoughts, behaviors, and/or body        sensations    -   1.4.2 Observes situations carefully    -   6.4.3 Implements problem solving strategies    -   6.5.3 Engages in Reflection (Code this micro if the student: 1)        Demonstrates that their experience included a reflective        process; 2) If student describes some kind of        emotional/reflective thinking response that came out of their        experience    -   1.2.2 Recognize one's shortcomings to overcome obstacles or        improve skills (e.g., I used to do X, but not doing that anymore        because it didn't serve)    -   5.5.3 Supports personal mental health

Example 3: Model Comparison

In example 3, three models were trained for predicting level-3micro-credentials: Logistic Regression, Nai{umlaut over (v)}e Bayes andSupport Vector Machines with linear kernel on a 5000 dimensional “termfrequency inverse document frequency” scores of the experience textcorpus that were received from applicants 104. A 10-foldcross-validation was conducted for each of the models, and theevaluation results are presented in Table 3, below. Each of the threeclassifiers show promising prediction performance where LogisticRegression based multi-output classification demonstrated slightlyfaster performance than the rest on an average.

TABLE 3 Experimental evaluation of three Bag-of-Word Models in terms ofaverage accuracy of each of predictions of 152 level-3 microcredentialclasses, and CPU time required for micro-credential assignment of eachexperience text submitted. Average Prediction Time Model Name AverageAccuracy per Submission Logistic Regression 0.96 ± 0.066 0.089959Support Vector Machines 0.96 ± 0.006 0.093059 Nai{umlaut over (v)}eBayes 0.94 ± 0.005 0.109604

Example 4: Unintentional Bias

A sample of output micro-credentials 122 were evaluated. When thepossible race categories were isolated to “white” and “black or AfricanAmerican” for the sake of illustration, 12.8% of the level-2micro-credentials have a difference in CSP greater than or equal to0.05. 60% of these differences favored white submissions. The annotatorsassessed these differences looking especially for initial annotationsthat might contain some annotator bias. For instance, Table 4 outlineseffectiveness of model refinement through the proposed iterativepipeline on a particular level-2 micro-credential, “Working withOthers”.

TABLE 4 Conditional statistical parity for the “Working with Others”credential by race, before and after annotation re-assessment. (Resultsfor other credentials will be included later for presentation.)Iteration White African American Before Annotation 0.1049 0.0368Reassessment After Annotation 0.1049 0.0743 Reassessment

After one iteration of training the model, the output micro-credentials122 were sent back to annotators 108 and asked them to reconsider theannotations for credentials exhibiting a difference in CSP value 136greater than or equal to 0.05 (e.g., CSP threshold 138) between twodemographic groups. This reconsideration was performed one demographicgroup at a time, so that the annotators 108 could consider possiblesystemic bias as explanations. For example, the annotators discoveredthat black or African American students in the sample tend to talk aboutworking with others passively, whereas white students tend to describeworking with others actively.

Referring to FIG. 1, accreditation portal 140 may further include adisplay module 140. Display module 140 interacts with applicant device110(1), accreditor device 110(2), and administrator device 110(3) todisplay information via one or more API calls/requests with the hostedURL browser on the given computing device 110. FIG. 17 depicts ascreenshot 1700 implemented by display module 140 showing a plurality ifmicro-credentials 122 gained 1702 and remaining needed 1704 to achievean accreditation credit 123, in an embodiment.

Display module 140 further includes various filters that are accessibleby applicant 104, accreditor 106, and administrator 108, via interactionwith their respective computing device 110. FIGS. 18-19, for example,show a screenshot 1800 and 1900, respectively allowing a user to filterfor experiences submitted (e.g., submissions 202, 204, or 206) within aselectable period 1902. FIG. 20 shows a screenshot 2000 of displayed andselectable experiences submitted after selection of one of theselectable periods 1802, of FIGS. 18-19, in an embodiment. FIG. 21 showsa screenshot 2100 displayed in response to user-selection of one of thedisplayed and selectable experiences shown in FIG. 20, in an embodiment.

It should be appreciated that, although FIGS. 18-21 show displaysassociated with the applicant device 110(1), similar displays may beconfigured by the display module 140 for each of the accreditor device110(2) and administrator device 110(3) to display and interact with eachof the accreditor 106 and administrator 108. For example, the accreditordevice 110(2) may display an interactive display interactable by theaccreditor 106 to receive configuration settings such as number ofmicro-credentials 122 required to achieve a given accreditation credit123, and which accreditation credits 123 are available to givenapplicants 104.

Changes may be made in the above methods and systems without departingfrom the scope hereof. It should thus be noted that the matter containedin the above description or shown in the accompanying drawings should beinterpreted as illustrative and not in a limiting sense. The followingclaims are intended to cover all generic and specific features describedherein, as well as all statements of the scope of the present method andsystem, which, as a matter of language, might be said to falltherebetween.

What is claimed is:
 1. A method for micro-credential accreditation,comprising: receiving a text submission from an applicant devicedescribing an event the applicant experienced; processing the textsubmission with a predictive model to fit at least one micro-credentialto the text submission, the micro-credential at least partiallyqualifying the applicant for credit from an accreditor; outputting theat least one micro-credential.
 2. The method of claim 1, whereinoutputting the at least one micro-credential includes providing schoolcredit to the applicant.
 3. The method of claim 1, wherein outputtingthe at least one micro-credential includes displaying, on the applicantdevice, a chart indicating required micro-credentials to receive adesired accreditation credit.
 4. The method of claim 1, the receiving atext submission comprising providing one or more prompts to theapplicant device, and receiving responses to said one or more promptsfrom the applicant device.
 5. The method of claim 4, the promptsincluding selectable prompts including a plurality of selectableoptions, and fillable prompts including an input box for receiving textinput.
 6. The method of claim 5, further comprising processing theresponses to generate a compiled submission, the compiled submissionincluding a text string, portions of the text string corresponding to aselected one of the selectable options compiled into a first portiontext string, and input text from the input box used as a second portionof the text string.
 7. The method of claim 1, further comprisinggenerating a processed submission by: removing gibberish text within thetext submission; standardizing text within the text submission; removingwhite space within the text submission; removing stop-words within thetext submission; stemming individual words or phrases within the textsubmission; and, performing lemmatization on the words or phrases withinthe text submission; wherein said processing the text submissionincludes processing the processed submission.
 8. The method of claim 1,further comprising generating a processed submission by identifying aterm frequency inverse document frequency (TD-IDF) matrix from thereceived text submission; wherein said processing the text submissionincludes processing the TD-IDF matrix.
 9. The method of claim 1, theprocessing the text submission comprising applying the text submission,or a processed version thereof, to a Pachinko Allocation Model to fitthe at least one micro-credential.
 10. The method of claim 1, theprocessing the text submission comprising applying the text submission,or a processed version thereof, to a plurality of different predictionmodels each generating a list of fit micro-credentials, and implementinga consensus algorithm to develop a final list of one or moremicro-credential from each least of fit micro-credentials.
 11. Themethod of claim 10, the consensus algorithm including one or both ofBoosting and stacked generalization.
 12. The method of claim 1, furthercomprising monitoring previously output micro-credentials by demographicgroups of applicants to identify whether the predictive model consistentbetween each demographic group; and updating the predictive model whenthe predictive model is not consistent within a fairness metric.
 13. Themethod of claim 12, wherein said monitoring comprises calculating aplurality of conditional statistical parity (CSP) values, each CSP valuedefining probability of a target micro-credential being issued to eachdemographic group; and comparing differences between CSP values to a CSPthreshold.
 14. A system for micro-credential accreditation, comprising:a processor; memory operatively coupled to the processor; amicro-credential predictive module defined as computer-readableinstructions within the memory and defining: a predictive model that,when the micro-credential predictive module is executed by theprocessor, fits a text submission to one or more targetmicro-credentials, and outputs the fit target micro-credentials asassigned micro-credentials.
 15. The system of claim 14, wherein thepredictive model includes a plurality of target classifiers, eachassociated with one or more of the target micro-credentials.
 16. Thesystem of claim 14, the target micro-credentials including a hierarchyof levels of micro-credentials.
 17. The system of claim 14, furthercomprising a model maintenance module that, when executed by theprocessor, identifies a fairness metric the one or more targetmicro-credentials.
 18. The system of claim 17, the fairness metric basedon conditional statistical parity of the one or more targetmicro-credentials between different demographic groups.
 19. A method formaintaining a predictive model used for micro-credential accreditation,comprising: outputting a plurality of micro-credentials, saidmicro-credentials being fit, from a group of target micro-credentials,based on a prediction model analyzing a plurality text submissionsreceived from one or more applicant devices; analyzing a fairness metricof the target micro-credentials by determining a likelihood that eachtarget micro-credential will be fit to each of a plurality ofdemographic groups; updating the prediction model when the fairnessmetric indicates that the prediction model does not output the targetmicro-credentials across the demographic groups within a fairnessthreshold.
 20. The method of claim 19, wherein analyzing the fairnessmetric comprises: calculating a plurality of conditional statisticalparity (CSP) values, each CSP value defining probability of a targetmicro-credential being issued to each demographic group; and comparingdifferences between CSP values to a CSP threshold.